Parallel Discrete Event Simulation Research Articles

Performance Evaluation of Spintronic-Based Spiking Neural Networks Using Parallel Discrete-Event Simulation

Spintronics devices that use the spin of electrons as the information state variable have the potential to emulate neuro-synaptic dynamics and can be realized within a compact form-factor, while operating at ultra-low energy-delay point. In this paper, we benchmark the performance of a spintronics hardware platform designed for handling neuromorphic tasks. To explore the benefits of spintronics-based hardware on realistic neuromorphic workloads, we developed a Parallel Discrete-Event Simulation model called Doryta, which is further integrated with a materials-to-systems benchmarking framework. The benchmarking framework allows us to obtain quantitative metrics on the throughput and energy of spintronics-based neuromorphic computing and compare these against standard CMOS-based approaches. Although spintronics hardware offers significant energy and latency advantages, we find that for larger neuromorphic circuits, the performance is limited by the interconnection networks rather than the spintronics-based neurons and synapses. This limitation can be overcome by architectural changes to the network. Through Doryta we are also able to show the power of neuromorphic computing by simulating Conway’s Game of Life (GoL), thus showing that it is Turing complete. We show that Doryta obtains over 300 × speedup using 1,024 CPU cores when tested on a convolutional, sparse, neural architecture. When scaled-up 64 times, to a 200 million neuron model, the simulation ran in 3:42 minutes for a total of 2000 virtual clock steps. The conservative approach of execution was found to be faster in most cases than the optimistic approach, even when a tie-breaking mechanism to guarantee deterministic execution, was deactivated.

An Axiomatic Theory for Reversible Computation

Undoing computations of a concurrent system is beneficial in many situations, e.g., in reversible debugging of multi-threaded programs and in recovery from errors due to optimistic execution in parallel discrete event simulation. A number of approaches have been proposed for how to reverse formal models of concurrent computation including process calculi such as CCS, languages like Erlang, and abstract models such as prime event structures and occurrence nets. However it has not been settled what properties a reversible system should enjoy, nor how the various properties that have been suggested, such as the parabolic lemma and the causal-consistency property, are related. We contribute to a solution to these issues by using a generic labelled transition system equipped with a relation capturing whether transitions are independent to explore the implications between various reversibility properties. In particular, we show how all properties we consider are derivable from a set of axioms. Our intention is that when establishing properties of some formalism it will be easier to verify the axioms rather than proving properties such as the parabolic lemma directly. We also introduce two new properties related to causal consistent reversibility, namely causal liveness and causal safety, stating, respectively, that an action can be undone if (causal liveness) and only if (causal safety) it is independent from all the following actions. These properties come in three flavours: defined in terms of independent transitions, independent events, or via an ordering on events. Both causal liveness and causal safety are derivable from our axioms.

Parallel Simulation of Quantum Networks with Distributed Quantum State Management

Quantum network simulators offer the opportunity to cost-efficiently investigate potential avenues for building networks that scale with the number of users, communication distance, and application demands by simulating alternative hardware designs and control protocols. Several quantum network simulators have been recently developed with these goals in mind. As the size of the simulated networks increases, however, sequential execution becomes time-consuming. Parallel execution presents a suitable method for scalable simulations of large-scale quantum networks, but the unique attributes of quantum information create unexpected challenges. In this work, we identify requirements for parallel simulation of quantum networks and develop the first parallel discrete-event quantum network simulator by modifying the existing serial simulator SeQUeNCe. Our contributions include the design and development of a quantum state manager (QSM) that maintains shared quantum information distributed across multiple processes. We also optimize our parallel code by minimizing the overhead of the QSM and decreasing the amount of synchronization needed among processes. Using these techniques, we observe a speedup of 2 to 25 times when simulating a 1,024-node linear network topology using 2 to 128 processes. We also observe an efficiency greater than 0.5 for up to 32 processes in a linear network topology of the same size and with the same workload. We repeat this evaluation with a randomized workload on a caveman network. We also introduce several methods for partitioning networks by mapping them to different parallel simulation processes. We have released the parallel SeQUeNCe simulator as an open-source tool alongside the existing sequential version.

Spatial/Temporal Locality-based Load-sharing in Speculative Discrete Event Simulation on Multi-core Machines

Shared-memory multi-processor/multi-core machines have become a reference for many application contexts. In particular, the recent literature on speculative parallel discrete event simulation has reshuffled the architectural organization of simulation systems in order to deeply exploit the main features of this type of machines. A core aspect dealt with has been the full sharing of the workload at the level of individual simulation events, which enables keeping the rollback incidence minimal. However, making each worker thread continuously switch its execution between events destined to different simulation objects does not favor locality. This problem appears even more evident in the case of Non-Uniform-Memory-Access (NUMA) machines, where memory accesses generating a cache miss to be served by a far NUMA node give rise to both higher latency and higher traffic at the level of the NUMA interconnection. In this article, we propose a workload-sharing algorithm where the worker threads can have short-term binding with specific simulation objects to favor spatial locality. The new bindings—carried out when a thread decides to switch its execution to other simulation objects—are based on both (a) the timeline according to which the object states have passed through the caching hierarchy and (b) the (dynamic) placement of objects within the NUMA architecture. At the same time, our solution still enables the worker threads to focus their activities on the events to be processed whose timestamps are closer to the simulation commit horizon—hence we exploit temporal locality along virtual time and keep the rollback incidence minimal. In our design we exploit lock-free constructs to support scalable thread synchronization while accessing the shared event pool. Furthermore, we exploit a multi-view approach of the event pool content, which additionally favors local accesses to the parts of the event pool that are currently relevant for the thread activity. Our solution has been released as an integration within the USE (Ultimate-Share-Everything) open source speculative simulation platform available to the community. Furthermore, in this article we report the results of an experimental study that shows the effectiveness of our proposal.

A parallel discrete event simulation engine for the low-earth-orbit satellite constellation networks

A parallel discrete event simulation engine for the low-earth-orbit satellite constellation networks

Parallel and Distributed Simulation of Ad Hoc Networks

Modeling and simulation are traditional methods used to evaluate wireless network design. This paper addresses issues associated with the application of parallel discrete event simulation to mobile ad hoc networks design and analysis. The basic characteristics and major issues pertaining to ad hoc networks modeling and simulation are introduced. The focus is on wireless transmission and mobility models. Particular attention is paid to the MobASim system, a Javabased software environment for parallel and distributed simulation of mobile ad hoc networks. We describe the design, performance and possible applications of presented simulation software.

Network accelerator for parallel discrete event simulations

The simulation time of parallel and distributed discrete event simulation is the heart rate for as-fast-as execution schemes. Agreeing upon a global simulation time and distributing it to the simulation processes can be improved by exploiting or redesigning the network hardware. In this paper, we present such an approach that offloads simulation time calculations to network switches in order to speed up the steps where time advance requests are made and time advance grants are waited. By reducing the waiting time for time advancement, we aim to improve the overall performance of the parallel simulations. The measurements from the FPGA-based hardware setup and the results from our network simulations show that overall performance can be improved when time management calculations are offloaded to the network switches. Additionally, the transient message problem is also solved within the network by not allowing the time control messages to bypass the time-dependent events. The network acceleration of the region-based event distribution is also studied, and offloading the region matching tasks to the network switches is found to be feasible to reduce the costs of node-based calculations, especially for fast-moving regions. In this study, we consider High Level Architecture (HLA) for simulation infrastructure and fat tree topology for high-performance networking.

A Parallel Algorithm to Accelerate DEVS Simulations in Shared Memory Architectures

We propose a new algorithm for the execution of Discrete Event System Specification (DEVS) simulations on parallel shared memory architectures. Our approach executes parallel discrete-event simulations by executing all tasks in the PDEVS simulation protocol in parallel. The algorithm works by distributing the computations among different cores on shared memory architectures. To show the benefits of our algorithm, we present the results of a set of experiments using a synthetic benchmark and a real-world scenario using two independent computer architectures. The results obtained show how our algorithm accelerates simulations up to eight times, improving previous approaches. In addition, we show that our approach scales when we increase the number of CPU-cores used.

Synchronous speculative simulation of tightly coupled agents in continuous time on CPUs and GPUs

Traditionally, parallel discrete-event simulations of agent-based models in continuous time are organized around logical processes exchanging time-stamped events, which clashes with the properties of models in which tightly coupled agents frequently and instantaneously access each other’s states. To illustrate the challenges of such models and to derive a solution, we consider the domain-specific modeling language ML3, which allows modelers to succinctly express transitions and interactions of linked agents based on a continuous-time Markov chain (CTMC) semantics. We propose synchronous optimistic synchronization algorithms tailored toward simulations of fine-grained interactions among tightly coupled agents in highly dynamic topologies and present implementations targeting multicore central processing units (CPUs) as well as many-core graphics processing units (GPUs). By dynamically restricting the temporal progress per round to ensure that at most one transition or state access per agent, the synchronization algorithms enable efficient direct agent interaction and limit the required agent state history to only a single current and projected state. To maintain concurrency given actions that depend on dynamically updated macro-level properties, we introduce a simple relaxation scheme with guaranteed error bounds. Using an extended variant of the classical susceptible-infected-recovered network model, we benchmark and profile the performance of the different algorithms running on CPUs and on a data center GPU.

Performance Analysis of Speculative Parallel Adaptive Local Timestepping for Conservation Laws

Stable simulation of conservation laws, such as those used to model fluid dynamics and plasma physics applications, requires the satisfaction of the so-called Courant-Friedrichs-Lewy condition. By allowing regions of the mesh to advance with different timesteps that locally satisfy this stability constraint, significant work reduction can be attained when compared to a time integration scheme using a single timestep size. However, parallelizing this algorithm presents considerable difficulty. Since the stability condition depends on the state of the system, dependencies become dynamic and potentially non-local. In this article, we present an adaptive local timestepping algorithm using an optimistic (Timewarp-based) parallel discrete event simulation. We introduce waiting heuristics to limit misspeculation and a semi-static load balancing scheme to eliminate load imbalance as parts of the mesh require finer or coarser timesteps. Last, we outline an interface for separating the physics of the specific conservation law from the temporal integration allowing for productive adoption of our proposed algorithm. We present a misspeculation study for three conservation laws, demonstrating both the productivity of the local timestepping API, for which 74% of the lines of code are reused across different conservation laws, and the robustness of the waiting heuristics—at most 1.5% of element updates are rolled back. Our performance studies demonstrate up to a 2.8× speedup versus a baseline unoptimized local timestepping approach, a 4x improvement in per-node throughput compared to an MPI parallelization of synchronous timestepping, and scalability up to 3,072 cores on NERSC’s Cori Haswell partition.

Virtual Time III, Part 1: Unified Virtual Time Synchronization for Parallel Discrete Event Simulation

Algorithms for synchronization of parallel discrete event simulation have historically been divided between conservative methods that require lookahead but not rollback, and optimistic methods that require rollback but not lookahead. In this paper we present a new approach in the form of a framework called Unified Virtual Time (UVT) that unifies the two approaches, combining the advantages of both within a single synchronization theory. Whenever timely lookahead information is available, a logical process (LP) executes conservatively using an irreversible event handler. When lookahead information is not available the LP does not block, as it would in a classical conservative execution, but instead executes optimistically using a reversible event handler. The switch from conservative to optimistic synchronization and back is decided on an event-by-event basis by the simulator, transparently to the model code. UVT treats conservative synchronization algorithms as optional accelerators for an underlying optimistic synchronization algorithm, enabling the speed of conservative execution whenever it is applicable, but otherwise falling back on the generality of optimistic execution. We describe UVT in a novel way, based on fundamental invariants, monotonicity requirements, and synchronization rules. UVT permits zero-delay messages and pays careful attention to tie-handling using superposition. We prove that under fairly general conditions a UVT simulation always makes progress in virtual time. This is Part 1 of a trio of papers describing the UVT framework for PDES, mixing conservative and optimistic synchronization and integrating throttling control.

Virtual Time III, Part 2: Combining Conservative and Optimistic Synchronization

This is Part 2 of a trio of works intended to provide a unifying framework in which conservative and optimistic synchronization for parallel discrete event simulations can be freely and transparently combined in the same logical process on an event-by-event basis. In this article, we continue the outline of an approach called Unified Virtual Time (UVT) that was introduced in Part 1 , showing in detail via two extended examples how conservative synchronization can be refactored and combined with optimistic synchronization in the UVT framework. We describe UVT versions of both a basic time windowing algorithm called Unified Simple Time Windows and a refactored version of the Chandy-Misra-Bryant Null Message algorithm called Unified CMB .

Adaptive total variation stable local timestepping for conservation laws

This paper proposes a first-order total variation diminishing (TVD) treatment for coarsening and refining of local timestep size in response to dynamic local variations in wave speeds for nonlinear conservation laws. The algorithm is accompanied with a proof of formal correctness showing that given a sufficiently small minimum timestep the algorithm will produce a TVD solution for nonlinear scalar conservation laws. A key feature of the algorithm is its formulation as a discrete event simulation, which allows for easy and efficient parallelization using existing software. Numerical results demonstrate the stability and adaptivity of the method for the shallow water equations. We also introduce a performance model to load balance and explain the observed performance gains. Performance results are presented for a single node on Stampede2's Skylake partition using an optimistic parallel discrete event simulator. Results show the proposed algorithm recovering 59%-77% of the theoretically achievable speed-up with the remainder being attributed to the cost of computing the CFL condition and load imbalance.

Parallel algorithm development and testing using Petri-object simulation

Parallel algorithms are problematic to develop because of the negative influence of synchronisation, complicated behaviour of threads’ capturing computing resources. Experimental results show performance time’s strong dependence on algorithm parameters, such as the number of subtasks and the complexity of each task. The optimal value of subtask complexity is revealed for the particular algorithm. It is the same for different complexity of the parallelised task (with the same computing resource). To guarantee algorithm speed-up it is important to have a method for investigating the efficiency of parallel algorithm before its implementation on specified computing resources. Stochastic Petri net potentially could be a high accuracy tool for investigating the efficiency of a parallel algorithm. However, a huge number of elements are needed to compose a model of non-trivial algorithm that limits the application of this tool in practice. Petri-object simulation method allows replication of Petri nets with specified parameters and model creation of a list of linked Petri-objects. Basic templates for the model creation of a multithreaded algorithm are developed. Applying these templates, the model of the parallel discrete event simulation algorithm is developed and investigated. By the model results, the algorithm parameters providing the least performance time can be determined.

Measuring the Realtime Capability of Parallel-Discrete-Event-Simulations

Speeding up Discrete Event Simulations (DESs) is a broad research field. Promising Parallel Discrete Event Simulation (PDES) approaches with optimistic and conservative synchronisation schemes have emerged throughout the years. However, in the area of real-time simulation, PDESs are rarely considered. This is caused by the complex problem of fitting parallel executed DES models to a real-time clock. Hence, this paper gives an extensive review of existing conservative and optimistic synchronisation schemes for PDESs. It introduces a metric to compare their real-time capabilities to determine whether they can be used for soft or firm real-time simulation. Examples are given on how to apply this metric to evaluate PDESs using synthetic and real-world examples. The results of the investigation reveal that no final answer can be given if PDESs can be used for soft or firm real-time simulation as they are. However, boundary conditions were defined, which allow a use-case specific evaluation of the real-time capabilities of a certain parallel executed DES. Using this in-depth knowledge and can lead to predictability of the real-time behaviour of a simulation run.

Local time evolution in Personal Communication Service model

We investigate the local time evolution in the Personal Communication Service (PCS) model simulated with the parallel discrete event simulation method’s optimistic algorithm. We propose a model for the optimistic local virtual time evolution (OLVT) in PCS, which is reminiscent of statistical physics’s surface growth. We use Rensselaer’s optimistic simulation system with the Time Warp implementation. We compare the results of the simulations of both PCS and OLVT models and found good agreement. We discuss the highlights of our approach in the analysis of scalability and synchronization using the OLVT model.

Scaled Static Analysis and IP Reuse for Out-of-Order Parallel SystemC Simulation

To preserve the SystemC semantics under parallel discrete event simulation, a compiler based approach statically analyzes race conditions in the design model. However, there are severe restrictions: the source code for the input design must be available in one file, which does not scale. This disables the use of intellectual property (IP) and hierarchical file structures. This paper scales the static analysis design flow to support separate files and IP reuse by introducing partial segment graph and partial port mapping abstractions and prevent IP security leakage. Experiments demonstrate the effective design flow and sustained speedup with parallel IPs.

An Adaptive Persistence and Work-stealing Combined Algorithm for Load Balancing on Parallel Discrete Event Simulation

Load imbalance has always been a crucial challenge in Parallel Discrete Event Simulation (PDES). In the past few years, we have witnessed an increased interest in using multithreading PDES on multi/many-core platforms. In multithreading PDES, migrating logical processes and coordinating threads are more convenient and cause lower overhead, which provides a better circumstance for load balancing. However, current algorithms, including the persistence-based scheme and work-stealing-based scheme, have their drawbacks. On one hand, persistence-based load balancers, which use the historical data to predict the future, will inevitably make some error. On the other hand, the work-stealing scheme ignores the application-related characteristic, which may limit the potential performance improvement. In this article, we propose an adaptive persistence and work-stealing combined dynamic load balancing algorithm (APWS). The algorithm detects load imbalance, adaptively rebalances the distribution of logical processes, and uses a greedy lock-free work-stealing scheme to eliminate bias at runtime. We assess the performance of the APWS algorithm by a series of experiments. Results demonstrate that our APWS algorithm achieves better performance in different scenarios.

A Distributed Shared Memory Middleware for Speculative Parallel Discrete Event Simulation

The large diffusion of multi-core machines has pushed the research in the field of Parallel Discrete Event Simulation (PDES) toward new programming paradigms, based on the exploitation of shared memory. On the opposite side, the advent of Cloud computing—and the possibility to group together many (low-cost) virtual machines to form a distributed memory cluster capable of hosting simulation applications—has raised the need to bridge shared memory programming and seamless distributed execution. In this article, we present the design of a distributed middleware that transparently allows a PDES application coded for shared memory systems to run on clusters of (Cloud) resources. Our middleware is based on a synchronization protocol called Event and Cross State Synchronization. It allows cross-simulation-object access by event handlers, thus representing a powerful tool for the development of various types of PDES applications. We also provide data for an experimental assessment of our middleware architecture, which has been integrated into the open source ROOT-Sim speculative PDES platform.

An efficient multi-threaded memory allocator for PDES applications

Parallel Discrete Event Simulation (PDES) aims at providing high performance solutions for simulation with large and complex models. Like traditional multi-threaded applications, PDES applications are also confronted with large overhead produced by memory allocation. Especially, complex models and frequent inter-thread communications in PDES applications will lead to large amounts of memory allocation and deallocation requests. Inter-thread communications are implemented by sending and receiving events based on shared memory. Hence, an efficient memory allocator has great impact on the performance of such applications. Current well-known allocators are dedicated to be scalable and lock-free. However, they ignore the difference between thread local and shared memory and manage them in the same way, which cannot make good use of their own characteristics. To solve this problem, this paper proposes a high efficient multi-threaded memory allocator named HMalloc, which innovatively separates thread local memory from shared memory. Thus, local memory allocation and deallocation will not lead to false sharing and lock contention. Moreover, coalescence free is applied to free local memory blocks. Furthermore, a flag-based shared memory management method is proposed to achieve lock-free shared memory allocation and deallocation. Experimental results show that HMalloc can achieve significant performance improvement in both traditional memory allocation benchmarks and typical PDES benchmarks when compared with existing well-known memory allocators.